Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ogcapi records - better "catalog.yaml" objects #115

Merged
merged 19 commits into from
Oct 16, 2024

Conversation

davidblasby
Copy link
Contributor

This improves JSON output for the landing page (/), /collections, and /collections/<collectionid>.

In the output of these I inject the OGCAPI Records catalog.yaml object.

A GN Portal (full portal or sub-portal) is an OGCAPI Catalog.

  1. Getting info

This will retrieve the GN ServiceRecord that's linked to a GN Source DB row. This can be done in the GN GUI: Admin Console-Settings-sources for sub-portals.

There is no GUI to set this for the "main" portal - do this in GN GUI: Admin Console-Settings-Settings-Catalog Service for the Web (CSW)-Record to use for GetCapabilities (system/csw/capabilityRecordUuid).

Basically, this PR will retrieve this linked record via the Elastic Index (JSON). This is processed in GN via the index-utils.xsl.

If there isn't a linked service record, then the catalog details will not be added (i.e. unchanged from how GN ogcapi records used to work).

  1. New Model Objects

I have put in a bunch of new model objects - OgcApi*. These are 1:1 with the OGCAPI object model - defined in the spec as well as the .yaml files. I've copy-and-pasted in the documentation from the .yaml files for readability.

NOTE: I also moved the old Link object to OgcApiLink for consistency.

  1. Landing page (/)

The landing page JSON also includes a new property systemInfo which contains a catalog.yaml object in it.
I'm not sure if this is allowed in the spec, but it allows for more metadata about the entire GN system ("ogcapi-records" server). This is useful for making a nicer landing page (cf. pygeoapi's landing page).

  1. A few details

Most of the work is done in ElasticIndexJson2CollectionInfo - this converts the Elastic Index JSON object to the ogcapi catalog.yaml object. There's also a simple test case in ElasticIndexJson2CollectionInfoTest.

  1. Example

Underlying XML linked Service Record (xml)
9bac358b-11ec-4293-aeef-5a077b778412.txt

Elastic Index (JSON):
elastic-index-record.txt

OGCAPI catalog.yaml output (JSON):
ogcpai-catalog.json

@davidblasby
Copy link
Contributor Author

Original PR is here - davidblasby#1

(reviewed by @josegar74 )

@fxprunayre - would love to talk to you about this during the sprint this week.

@davidblasby
Copy link
Contributor Author

added 2 commits;

  1. update CORS to allow connections
    This shouldnt be a problem because the ogcapi is all GET-based (readonly)
  2. add collection (catalog/sub-portal) icon

@davidblasby
Copy link
Contributor Author

davidblasby commented Oct 1, 2024

@josegar74 @fxprunayre @f-necas I'd like to merge this is this week. I'm working on Queryables and Search right now (#116), and want to finish this up.

There is a Sonar warning - this is because I set the CORS so the ogcapi-records is accessible as a service. They are all GETs at the moment, so i don't think this is too big an issue.

@davidblasby davidblasby mentioned this pull request Oct 1, 2024
@josegar74
Copy link
Member

josegar74 commented Oct 7, 2024

@davidblasby If the metadata has online resources, the geojson output fails with 406 error. for example with the sample metadata Hydrological Basins in Africa (Sample record, please remove!)

http://localhost:9901/collections/main/items/da165110-88fd-11da-a88f-000d939bc5d8

I see in the log:

com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "hash" (class org.fao.geonet.index.model.gn.Link), not marked as ignorable (9 known properties: "nilReason", "protocol", "applicationProfile", "nameObject", "descriptionObject", "mimeType", "group", "function", "urlObject"])
 at [Source: (String)"{
  "docType" : "metadata",
  "document" : "",
  "metadataIdentifier" : "da165110-88fd-11da-a88f-000d939bc5d8",

Apparently some issue with the index model in GeoNetwork adn the class that models links in this project.

Recently was added a hash property in the links.

@davidblasby to solve this issue, please update https://github.com/geonetwork/geonetwork-microservices/blob/main/modules/library/common-index-model/src/main/java/org/fao/geonet/index/model/gn/Link.java

And add:

  private String hash;
  private String idx;

@josegar74
Copy link
Member

About

4. Landing page (/)
The landing page JSON also includes a new property systemInfo which contains a catalog.yaml object in it.
I'm not sure if this is allowed in the spec, but it allows for more metadata about the entire GN system ("ogcapi-records" server). This is useful for making a nicer landing page (cf. pygeoapi's landing page).

You mean that pygeoapi's uses this systemInfo also? or it's completely custom to GeoNetwork implementation?

We should check if there is a validator available, but last time I checked some months ago, that was not the case.

@davidblasby
Copy link
Contributor Author

You mean that pygeoapi's uses this systemInfo also? or it's completely custom to GeoNetwork implementation?

No - Its unclear where pygeoapi gets its information from (its not in its JSON output).
there is no systeminfo inside pygeoapi. I just added it so an external (i.e. angular) app would have access to this info.

@davidblasby
Copy link
Contributor Author

@josegar74 - added the hash/idx.

@davidblasby
Copy link
Contributor Author

I updated this to use the parsed IndexRecord instead directly from the elastic index json.

This is in prep for GeoJSON changes.

Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarCloud

@davidblasby
Copy link
Contributor Author

@josegar74 @fxprunayre - will merge this tomorrow.

@fxprunayre
Copy link
Member

Sounds fine to add this @davidblasby. There is some issues
http://localhost:9901/collections/inspire?f=json will return service info
http://localhost:9901/collections/inspire?f=xml will not (related to getRecordAsJson and request headers) but for demo purpose that can be ok.

Also we should plan to move it to GeoNetwork 5 codebase at some point

@davidblasby
Copy link
Contributor Author

@fxprunayre - I havent been looking at the XML output - just the JSON. I will put that on the list!

Whats involved with moving to the GN5 codebase? I don't know anything about it...

@davidblasby davidblasby merged commit a254360 into geonetwork:main Oct 16, 2024
1 of 2 checks passed
@fxprunayre
Copy link
Member

@fxprunayre - I havent been looking at the XML output - just the JSON. I will put that on the list!

Not urgent.

Whats involved with moving to the GN5 codebase? I don't know anything about it...

We are cleaning up the POC with @josegar74 and we will move the repository to GeoNetwork organisation as discussed in Bennekom. Then we can look into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants